Adaptive image browsing

ABSTRACT

A method for browsing a collection of multimedia assets on a soft-copy display comprising: receiving a collection of multimedia assets, each multimedia asset having a set of associated metadata; defining an interest criterion; using a processor to analyze the metadata associated with the multimedia assets and rank the multimedia assets according to the interest criterion; displaying a portion of the multimedia assets on the soft-copy display according to the ranking; interactively user selecting a displayed multimedia asset using a user interface; updating the interest criterion based on the selected multimedia asset; using a processor to determine an updated ranking of the multimedia assets according to the updated interest criterion; and displaying a portion of the multimedia assets on the soft-copy display according to the updated ranking.

CROSS-REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned, co-pending Great Britain PatentApplication Serial No. 0818089.5 (PCT/EP2009/060451), entitled:“Interactive Image Selection Method”, by Fry et al., which isincorporated herein by reference.

FIELD OF THE INVENTION

This invention pertains to the field of browsing collections ofmultimedia assets and more particularly to a method for adaptivelyupdating an interest criteria used to create a ranked display.

BACKGROUND OF THE INVENTION

The problem with standard “search and retrieve” techniques when appliedto large image collections is the underlying assumption that users “knowwhat they are looking for” and that all of the images have been taggedwith meaningful key words. This may be the case in certain situations,but more likely, users browse image collections as a form ofentertainment or to reminisce and seldom take the time an effort toprovide key word information. Images are typically used as an adjunct torecalling past events, enhancing social connections, or establishing newrelationships where the element of discovery is essential in making theexperience enjoyable.

Designating “Favorites” was an early attempt to establish that certainimages, in a multimedia collection, had special significance to theuser. This feature could only be used to illustrate that an image orgroup of images are special but does not address why they are special orprovide any indications about the relationships between images image ina collection other than they are the user's “favorites” or “notfavorites”. Favorite status is manually assigned by a user at the timeof image capture via suitably equipped cameras or while viewing imageson a display device with software that provides this capability. In bothcases, the user is required to make the designation manually and asmultimedia content collections have grown in size, designating favoriteshave become increasingly less useful.

Another popular method of assigning special significance to multimediacontent is to assign selected content to virtual “albums”. Albums haveuser designated or selected themes or names that attempt to reflect thetype of content associated with the album. Some systems that employalbums store content files in a folder structure within a device orassessable by a device. Other content storage and presentation systemsusing albums only point to the content files so that the files do notneed to be physically rearranged within a storage device and can beincluded in multiple albums without requiring redundant storage. Bothtypes of album systems require the user to make decisions and manualdesignations that become increasingly burdensome for casual users asimage collections have grown is size.

Another method of accessing multimedia content collections is by usingsearching algorithms. Text based searches require that keywords havealready been assigned to images and tend to focus people down tospecific images or categories of images, where the activity gets stuck,not necessarily at a point the user wanted to get to. It also reducesthe possibility of serendipitous discovery and side tracking behavior.“Search and retrieve” techniques often makes interacting with imagesfeel more like work and less like the enjoyable browsing type ofbehavior people typically undertook with a shoebox of images, wherebeing sidetracked was part of the entertaining experience.

U.S. Patent Application Publication No. 2009/0094518 to Lawther et al.,entitled “Method For Image Animation Using Image Value Rules,” teaches amethod for image presentation, where the display is animated in a mannerto attract attention to images determined to have a higher image valuescore. No provision is made for adjusting the images that are includedin the image presentation according to a user-specified interestcriterion in order to enable an active browsing experience.

-   -   European Patent Application 1089198 Wong entitled “Content-Based        Image Searching” discloses a hierarchical method to derive a        content-based image tree from an image database. The method        partitions the set of multiple images into a first subset and a        second subset by allocating images to each subset whose feature        metrics are closer to a feature metric of a seed image in each        respective subset. For each subset, a comparison image is        determined whose feature metric is representative of feature        metrics of images partitioned into the respective subsets. This        process is recursively repeats for each subset, and for subsets        of each subset, etc., to create a tree of partitioned subsets of        images. The method then enables a search based on the comparison        images.    -   International Patent Application WO 2008/073505 to Marques et        al. entitled “System and Methods of Image Retrieval” discloses a        visual image retrieval system. A preprocessor communicatively        linked to an image database segments the images and determines a        regions-of-interest. A perceptually-relevant image search        machine interface is used to display query images and retrieve        images in response to a user selecting at least one displayed        query image.    -   International Patent Application WO 2005/008519 to Zheng et al.        entitled “Combined Search Method for Content-Based Image        Retrieval” discloses a method for performing combined image        classification, storage and rapid visual content retrieval on a        computer database. The method uses random combinatory        information and involves sorting relevant information into        categories and organizing it into data structures according to        allocated indices.    -   International Patent Application WO 2004/061710 to Stauder et        al. entitled “Process and Device for Access to Multimedia        Elements by Means of Images Associated with These Elements”        discloses a process for accessing multimedia elements. To access        one of these elements, a first group of images associated with a        first element is displayed, and means is provided for displaying        a second group of images associated with at least one of these        displayed images.

SUMMARY OF THE INVENTION

The present invention represents a method for browsing a collection ofmultimedia assets on a soft-copy display comprising:

a) receiving a collection of multimedia assets, each multimedia assethaving a set of associated metadata;

b) defining an interest criterion;

c) using a processor to analyze the metadata associated with themultimedia assets and rank the multimedia assets according to theinterest criterion;

d) displaying a portion of the multimedia assets on the soft-copydisplay according to the ranking;

e) interactively user selecting a displayed multimedia asset using auser interface;

f) updating the interest criterion based on the selected multimediaasset;

g) using a processor to determine an updated ranking of the multimediaassets according to the updated interest criterion; and

h) displaying a portion of the multimedia assets on the soft-copydisplay according to the updated ranking.

This invention has the advantage that it provides an intuitive processfor a user to browse a large collection of multimedia assets to identifymultimedia assets satisfying an interactively-updated interestcriterion. As the interest criterion evolves during the browsingprocess, the image display is updated to include new multimedia assetsthat reflect the updated interest criterion.

Another advantage of the invention is that it can take into account thenumber of similarly ranked related multimedia assets and limit thenumber of multimedia assets that are presented to the user in order todisplay other lower-ranked multimedia assets that that still may be ofinterest to the user.

It has the additional advantage of supporting multiple presentationformats enabling user selectable interaction options to adapt to thepreferences of individual users.

It has the further advantage that the inclusion of randomly-selectedmultimedia asset provide a mechanism for the user to encouragesidetracking, thus providing a more serendipitous experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram showing the components of a system forbrowsing digital images according to an embodiment of the presentinvention;

FIG. 2 is a flow chart illustrating the process steps for classifyingdigital images according to an embodiment of the present invention;

FIG. 3 is a graphical representation depicting a spiral formation ofmultimedia assets arranged in rank order together with a randomselection of multimedia assets according to an embodiment of the presentinvention;

FIG. 4 is a graphical representation depicting a spiral formation ofmultimedia assets arranged in a first modified rank order;

FIG. 5 is a graphical representation depicting a spiral formation ofassets arranged in a second modified rank order,

FIG. 6 is a graphical representation depicting rank-ordered multimediaassets arranged in a matrix format according to an embodiment of thepresent invention;

FIG. 7 is a graphical representation depicting rank-ordered multimediaassets with a “film strip” preview selector according to an embodimentof the present invention; and

FIG. 8 is a graphical representation depicting rank-ordered multimediaassets in a list presentation format according to an embodiment of thepresent invention.

It is to be understood that the attached drawings are for purposes ofillustrating the concepts of the invention and may not be to scale.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, some embodiments of the present inventionwill be described in terms that would ordinarily be implemented assoftware programs. Those skilled in the art will readily recognize thatthe equivalent of such software may also be constructed in hardware.Because image manipulation algorithms and systems are well known, thepresent description will be directed in particular to algorithms andsystems forming part of, or cooperating more directly with, the methodin accordance with the present invention. Other aspects of suchalgorithms and systems, together with hardware and software forproducing and otherwise processing the image signals involved therewith,not specifically shown or described herein may be selected from suchsystems, algorithms, components, and elements known in the art. Giventhe system as described according to the invention in the following,software not specifically shown, suggested, or described herein that isuseful for implementation of the invention is conventional and withinthe ordinary skill in such arts.

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to the “method” or “methods” and thelike is not limiting. It should be noted that, unless otherwiseexplicitly noted or required by context, the word “or” is used in thisdisclosure in a non-exclusive sense.

The phrase, “multimedia asset”, as used herein refers to any type ofmultimedia asset such as digital image files, digital video files anddigital audio files.

FIG. 1 is a high-level diagram showing the components of a system foraccessing and viewing multimedia collections according to an embodimentof the present invention. The system includes a data processing system110, a peripheral system 120, a user interface system 130, and a datastorage system 140. The peripheral system 120, the user interface system130 and the data storage system 140 are communicatively connected to thedata processing system 110.

The data processing system 110 includes one or more data processingdevices that implement the processes of the various embodiments of thepresent invention, including the example processes described herein. Thephrases “data processing device” or “data processor” are intended toinclude any data processing device, such as a central processing unit(“CPU”), a desktop computer, a laptop computer, a mainframe computer, anetbook, a personal digital assistant, a Blackberry™, a digital camera,cellular phone, or any other device for processing data, managing data,or handling data, whether implemented with electrical, magnetic,optical, biological components, or otherwise.

The data storage system 140 includes one or more processor-accessiblememories configured to store information, including the informationneeded to execute the processes of the various embodiments of thepresent invention, including the example processes described herein. Thedata storage system 140 may be a distributed processor-accessible memorysystem including multiple processor-accessible memories communicativelyconnected to the data processing system 110 via a plurality of computersor devices. On the other hand, the data storage system 140 need not be adistributed processor-accessible memory system and, consequently, mayinclude one or more processor-accessible memories located within asingle data processor or device.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, floppy disks, hard disks, Compact Discs,DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data may be communicated. The phrase“communicatively connected” is intended to include a connection betweendevices or programs within a single data processor, a connection betweendevices or programs located in different data processors, and aconnection between devices not located in data processors at all. Inthis regard, although the data storage system 140 is shown separatelyfrom the data processing system 110, one skilled in the art willappreciate that the data storage system 140 may be stored completely orpartially within the data processing system 110. Further in this regard,although the peripheral system 120 and the user interface system 130 areshown separately from the data processing system 110, one skilled in theart will appreciate that one or both of such systems may be storedcompletely or partially within the data processing system 110.

The peripheral system 120 may include one or more devices configured toprovide digital content records to the data processing system 110. Forexample, the peripheral system 120 may include digital still cameras,digital video cameras, cellular phones, or other data processors. Thedata processing system 110, upon receipt of digital content records froma device in the peripheral system 120, may store such digital contentrecords in the data storage system 140.

The user interface system 130 may include a mouse, a keyboard, anothercomputer, or any device or combination of devices from which data isinput to the data processing system 110. In this regard, although theperipheral system 120 is shown separately from the user interface system130, the peripheral system 120 may be included as part of the userinterface system 130.

The user interface system 130 also may include a display device, aprocessor-accessible memory, or any device or combination of devices towhich data is output by the data processing system 110. In this regard,if the user interface system 130 includes a processor-accessible memory,such memory may be part of the data storage system 140 even though theuser interface system 130 and the data storage system 140 are shownseparately in FIG. 1.

The present invention will now be described with reference to FIG. 2,which is a flow chart illustrating the process steps for browsingdigital images. A rank multimedia assets step 204 is used to rank amultimedia asset collection 200 according to an interest criterion 202provided by a user to produce a ranked multimedia asset collection 206.A display portion of multimedia collection step 208 produces a set ofdisplayed multimedia assets 210, which is a subset of the rankedmultimedia asset collection 206. Next, a user interface is provided toenable a user to identify a selected multimedia asset 214 using a selectmultimedia asset step 212. An update interest criterion step 216 is usedto update the interest criterion 202 responsive to the selectedmultimedia asset 214, producing an updated interest criterion 218. Theupdated interest criterion 218 is then used by the rank multimediaassets step 204 to produce a new ranked multimedia asset collection 206,which is then used to produce an updated display. This process can beperformed iteratively as desired by the user.

Each of the steps in the method of FIG. 2 will now be described in moredetail. The multimedia asset collection 200 can contain a collection ofdigital images, digital video files, digital audio files, and otherforms of multimedia content. The multimedia asset collection 200 can bemultimedia assets owned by a particular user, or can include multimediaassets owned by a plurality of users. The multimedia asset collection200 can be stored on one particular data storage system, or can includemultimedia assets stored on a variety of different data storage systemsto which the user has access. For example, images can be stored onservers that are accessible over the Internet or dispersed amongmultiple users' systems as with a peer-to-peer network.

The interest criterion 202 can be provided by text input from the user,by selection from a list of predefined criteria, from informationobtained from a user profile, or by selecting a particular multimediaasset of interest to the user. The interest criterion 202 can relate toevents, locations, peoples, time-frames, anniversaries, specialoccasions, or the visual appeal of an asset. For example, a user mayspecify that he is interested in viewing images of a particular personby typing the persons name into a search query, by selecting a name froma list of all known individuals in the multimedia asset collection, orby selecting an image containing the person of interest. If the userfails to provide an initial interest criterion 202, a default interestcriteria can be provided. For example, a default interest criteria canbe provided based on an interest criterion used in a previous session,or based on a generic interest criteria such as images having the mostrecent capture time.

The multimedia assets in the multimedia asset collection 200 arerank-ordered in rank multimedia assets step 204 from most to leastrelevant according to the specified interest criterion 202. The rankmultimedia assets step 204 can use any method for ranking objectsaccording to some criterion known to one skilled in the art. In oneembodiment of the present invention, metadata associated with themultimedia assets is used to rank the multimedia assets using anyranking algorithm known to one skilled in the art.

Metadata useful in the ranking process is often stored as metadata tagsassociated with the multimedia asset file. Such metadata typicallyincludes information obtained from the original capture device, such astime, date or location of image capture, camera settings, and otherinformation associated with the device. It can also include metadataadded by a user at the time of image capture, or at some later time.

Additional metadata can also be obtained by semantic or salient analysisof the multimedia asset. Such analysis can be used to determine manytypes of metadata including event type, number and size of faces in theimage, identity of individuals in an image, objects contained in theimage, distribution of colors in the image, similarity between images,and the general appeal of an image as determined by techniques includingviewing dwell time and or frequency of viewing.

Examples of methods for automatically analyzing multimedia assets usingsemantic or salient analysis algorithms to obtain metadata useful forthe rank multimedia assets step 204 would include the following. U.S.Pat. No. 6,847,733 to Savakis, entitled “Retrieval and Browsing ofDatabase Images Based on Image Emphasis and Appeal” and U.S. Pat. No.6,671,405 to Savakis, entitled “Method for Automatic Assessment ofEmphasis and Appeal in Consumer Images,” teach methods for determiningthe degree of importance, interest or attractiveness of an image.

U.S. Pat. No. 6,282,317 to Luo et al., entitled “Method for AutomaticDetermination of Main Subjects in Photographic Images,” discloses amethod for detecting a main subject in an image that includesidentifying at least one structural saliency feature and at least onesemantic saliency feature.

U.S. Pat. No. 6,804,684 to Stubler, entitled “Method for AssociatingSemantic Information with Multiple Images in an Image DatabaseEnvironment,” discloses a method of generating captions or semanticlabels for an acquired image based upon similarity between the acquiredimage and one or more stored images.

U.S. Pat. No. 6,826,316 to Luo et al., entitled “System and Method forDetermining Image Similarity,” discloses a system and method fordetermining image similarity using perceptually significant featuressuch as color, texture and shape.

U.S. Pat. No. 7,555,165 to Luo, entitled “Method for Semantic SceneClassification using Camera Metadata and Content-Based Cues,” disclosesa method for scene classification that includes extractingpre-determined camera metadata tags from the digital image and obtainingestimates of image class based on the extracted metadata tags.

U.S. Pat. No. 7,003,139 to Endrikhovski et al., entitled “Method forusing Facial Expression to Determine Affective Information in an ImagingSystem,” and U.S. Pat. No. 7,233,684 to Fedorovskaya et al., entitled“Imaging Method and System Using Affective Information,” disclosemethods for determining affective information for at least one image bymonitoring facial expression of a user as the user views the digitalimage.

U.S. Pat. No. 7,271,809 to Fedorovskaya et al., entitled “Method forUsing Viewing Time to Determine Affective Information in an ImagingSystem” discloses a method for determining affective information for atan image based on monitoring the viewing time.

U.S. Pat. No. 7,620,270 to Matraszek et al., entitled “Method forCreating and Using Affective Information in a Digital Imaging System”discloses determining a numeric categorization of images based onaffective information.

In a preferred embodiment of the present invention, the well-knownDempster-Shafer combination algorithm is used for the ranking algorithm.(For more information on using the Dempster-Shafer combination algorithmto rank images, see the article by J. Urban et al. entitled “An Adaptivetechnique for content-based image retrieval” published in the journalMultimedia Tools and Applications, Vol. 31, pp. 1-28, 2006.) Thisapproach combines evidence from different sources to arrive at a degreeof belief, represented by a belief function, taking into account all theavailable evidence. For example, in the case that the initial interestcriteria 202 specified that the user is interested in images of aparticular person, the ranking algorithm would rank images containingthat person more highly than those not containing the individual. In oneembodiment of the present invention, images where the individual isalone would be ranked more highly than group shots containing thatperson, and images where the individual fills the image would be rankedmore highly than images where the individual is standing in thebackground. Similarly, other factors can also be used to rank theimages. For example, images that are determined to have high sharpness,high colorfulness and good contrast using objective analysis algorithmscould be ranked higher than other images. Similarly, image that areanalyzed using automatic algorithms and determined to have a high levelof image value or general appeal, or images that were taken at a similartime and location to a highly ranked image could be ranked higher thanother images.

The display portion of multimedia collection step 208 displays a portionof the ranked multimedia asset collection 206. Generally, the size orposition of the displayed multimedia assets 210 are adjusted accordingto the ranking. In a preferred embodiment of the present invention, thedisplayed multimedia assets 210 are arranged in an ordered sequence, andwherein the highest ranked multimedia assets are displayed at the startof the ordered sequence.

Generally, a particular user interface may be limited to displaying acertain number of images. For example, if an image collection contains1000 images, the user interface may be limited to displaying only asmall fraction of those images (e.g., 30). In one embodiment of thepresent invention, the display portion of multimedia collection step 208selects the most highly ranked multimedia assets in the rankedmultimedia asset collection 206 as the displayed multimedia assets 210.

In a preferred embodiment of the present invention, the displayedmultimedia assets 210 are displayed on a softcopy display as still ormoving images, thumbnail images, or icons in patterns that intuitivelyreflect the rank status of the multimedia assets. As will be discussedin more detail later, there are many types of user interfaces that canbe used to display the displayed multimedia assets 210. Various designsinclude spirals, matrices, “film strip” metaphors, lists, and the like.In some embodiments, the relative size of the displayed multimediaassets 210 can be modified to reflect the rank order status. Forexample, more highly ranked multimedia assets can be displayed with alarger size. Other special indicators such as graphical treatments,borders, colors, icons, or indicators can also be used as an indicationof the relative rank of the multimedia assets.

In some cases, there may be a large number of highly ranked images inthe ranked multimedia asset collection 206. For example, if the interestcriteria simply specified that the user is interested in images thatcontain a particular individual, there may be a large number of imagesthat contain that individual. In this case, it may be desirable to makesure that the displayed multimedia assets include a variety of differentimages. Therefore, rather than simply selecting the most highly rankedmultimedia assets to be the displayed multimedia assets 210, the displayportion of multimedia collection step 208 may take other factors intoaccount. For example, it might randomly select images that have aranking greater than some threshold value. Alternately, it might limitthe number of images that are displayed that have a similar appearanceor were captured at a similar time or location. In this way, it ispossible to avoid the situation where all of the displayed multimediaassets 210 correspond to a single event, such as with a safari holidaythat had been documented with hundreds of still images and videorecordings.

Optionally, it may also be desirable to display a sampling ofloosely-related or unrelated digital multimedia assets to provide amechanism for the user to explore other parts of the multimedia assetcollection as part of the image browsing process. For example,loosely-related multimedia assets that are somewhat lower ranked, butshare one or more common metadata elements with a highly rankedmultimedia asset (e.g., images captured at the same event as a highlyranked image) can be included in the displayed multimedia assets. In oneembodiment of the present invention, the loosely-related multimediaassets can be displayed in the user interface near the end of the rankedsequence of images.

Similarly, unrelated multimedia assets can also be included in thedisplayed multimedia assets 210. In one embodiment of the presentinvention, the unrelated multimedia assets are included in the userinterface in such a way that they can be clearly identified as beingunrelated. For example, they can be scattered around the edge of thedisplay window.

The inclusion of loosely-related and unrelated multimedia assets offersthe user the option to focus in on a very specific event at a particulartime, while at the same time offering other options to encouragesidetracking, thus providing a more serendipitous experience thatattempts to emulate the established behavior of browsing through ashoebox of photographs with varying levels of organization.

The select multimedia asset step 212 can be carried out using any typeof user interface known to one skilled in the art that will allow a userto select of the displayed multimedia assets 210. For example, theselection process can take the form of touching an asset of interestdisplayed on a touch screen display, using a pointing device (e.g., amouse, a trackball, a keyboard, haptic game controller, or a tabletinterface) and graphical icon to point to the asset, or using voice orgesture commands to select the multimedia asset of interest. The userhas the option to select one of the highly-ranked multimedia assets, oroptionally to select one of the loosely-related or unrelated multimediaassets.

At the update interest criterion step 216 the selected multimedia asset214 is used to update the interest criterion 202, forming an updatedinterest criterion 218. In a preferred embodiment of the presentinvention, the interest criterion is updated by adding an additionalfactor that provides additional weight to multimedia assets that aresimilar to the selected multimedia asset. For example, the interestcriterion can be updated to emphasize multimedia assets containingpersons contained in selected multimedia asset, capture locations nearto the selected multimedia asset or capture dates similar to theselected multimedia asset.

The interest criterion 202 is then updated to be equal to the updatedinterest criterion 218, and is then used by the rank multimedia assetsstep 204 to provide a new ranked multimedia asset collection 206. In apreferred embodiment of the present invention, the previously selectedmultimedia asset 214 will now be the most highly-ranked multimediaassets, and multimedia assets that are similar to the selectedmultimedia asset 214 will be more highly ranked than those that aredissimilar. The similarity could be relative to image appearance asdetermined by an image similarity algorithm, or alternately could bewith respect to other aspects such as the image content, the number ofpeople in the image, or the capture time and location. For Example, ifthe original interest criterion was specified to be images containing aparticular individual and the user chooses a selected multimedia assetof that individual standing by the grand canyon with his wife, thehighly-ranked images in the updated ranked multimedia collection 208would include other images of the individual and his wife, as well asother images of the individual at the Grand Canyon, other images of theindividual captured on that day and other pictures of the individual inoutdoor locations.

The updated ranked multimedia asset collection 206 and then be used toselect and display an updated set of displayed multimedia assets 210. Asbefore, the displayed multimedia assets can include lower-rankedmultimedia assets such as loosely-related multimedia assets or randomlyselected multimedia assets. In one embodiment of the present invention,loosely-related multimedia assets are selected that share one or moreattributes in common with the selected multimedia asset. For example,the one or more common attributes can include similar capture time,similar capture location, similar appearance, similar audiocharacteristics, or one or more common persons, objects, events oractivities.

The user can then select a new multimedia asset and the process can becontinued iteratively. With each successive iteration, the updatedinterest criterion 218 can include the factors from the previousinterest criterion, together with the added criterion that thehighly-ranked multimedia assets should be similar to the most recentlyselected multimedia asset 214. In a preferred embodiment of the presentinvention, the weight given to the most recently selected multimediaasset 214 is given a high weight, which the weight given to thepreviously selected multimedia assets decreases with each successiveiteration.

This method of the present invention allows a user to hone in on imagesof interest. For example, if a user repeatedly selects images from thesame location, the algorithm can increasing favor location as a measure,thus increasingly its weight in the ranking process and moving imagescaptured in the same location higher in the rankings. This can allow theuser to focus down into an area of the collection. However, to preventthe user from getting caught in a dead end, the method enables the userto branch off in new directions by continuing to display a selection ofloosely-related and unrelated multimedia assets. Thus, the method of thepresent invention can respond in real time to a user's evolving interestwhich can permit the user follow a sidetrack into a completely differentarea of the collection.

Referring now to FIG. 3, a graphical representation is shown depictingan initial spiral formation of multimedia assets 222 on a display screen220. The initial spiral formation of multimedia assets 222 is arrangedin a rank order in response to an initial interest criterion. Thedisplay screen 220 can be any type of display screen known to thoseskilled in the art, including color LCD, plasma, OLED, CRT, lenticular,barrier, projection, 3D or volumetric type displays. The display screen220 can optionally include a touch sensitive surface to provide a touchscreen interface. The display screen 220 is connected to a stationary ora portable device capable of providing multimedia content and respondingto user interactions.

Within the initial spiral formation of multimedia assets 222, thelargest of the images at the upper right end of the spiral formationrepresents the highest ranked multimedia asset 223. The lower-rankedmultimedia assets are arranged sequentially around the spiral formation,with lower-ranked multimedia assets having a smaller size thanhigher-ranked multimedia assets, and being positioned in the backgroundbehind the higher-ranked multimedia assets. Other optional features notshown in FIG. 3 can also be included in the image display consistentwith conventional content presentation system, such as menus and userinterface controls for zooming and scrolling.

A selection of random multimedia assets 224 are also shown in a randomarrangement scattered around the periphery of the spiral formationaccording to an embodiment of the present invention. The presentation ofrandom multimedia assets 224 is optional and in a preferred embodimentof the present invention is a user selectable option that can be turnedon or off depending on the type browsing experience in which the userwishes to engage.

According to the present invention, a user interface is provided thatallows the user to select one of the displayed multimedia assets usingthe select multimedia asset step 212 (FIG. 2). For example, the user canuse a mouse or some other pointing device to click on a particulardisplayed multimedia asset, or can use a touch screen interface to toucha particular displayed multimedia asset. In this example, the userselects a selected multimedia asset 226, which is used by the updateinterest criterion step 216 (FIG. 2) to form an updated interestcriterion 218 (FIG. 2).

Referring now to FIG. 4, a graphical representation is shown depictingan updated spiral formation of multimedia assets 228 arranged in firstmodified rank order in response to the updated interest criterion 218.The selected multimedia asset 226 which was selected by the user in FIG.3 is now in the foreground and is positioned as the highest rankedmultimedia asset in the spiral formation. In addition to the updatedspiral formation of multimedia assets 228 a new selection of randommultimedia assets 224 is also presented on display screen 220 accordingto an embodiment of the present invention. As discussed earlier, therandom multimedia assets 224 offer opportunities to the user forside-tracking, thus enabling the user to jump in a new randomly chosendirection within the multimedia collection. In this example, the userchooses a new selected multimedia asset 230 from the displayed randommultimedia assets 224. An updated interest criterion 218 (FIG. 2) isthen determined responsive to this selection, and a new ranking for themultimedia asset collection is determined accordingly.

Referring now to FIG. 5 a graphical representation is shown depicting anupdated spiral formation of multimedia assets 232 arranged in secondmodified rank order in response to the updated interest criterion 218.The selected multimedia asset 230 which was selected by the user in FIG.4 is now in the foreground and is positioned as the highest rankedmultimedia asset in the spiral formation. The highest-ranked multimediaassets in the updated spiral formation of multimedia assets 232 arethose that are determined to be similar to the selected multimedia asset230. The updated spiral formation of multimedia assets 232 also includesa group of maintained multimedia assets 234 that were highly rankedaccording to the previous interest criterion in order to maintain ahistory of interest criterion used to get to the current manifestationof the asset collection. This reflects the fact that the update interestcriterion step 216 still includes the previously selected multimediaassets at a lower weight. In one embodiment of the present invention,the most highly-ranked multimedia assets in the previous iteration aredisplayed whether or not they end up being highly ranked according tothe updated interest criterion. The presence of the maintainedmultimedia assets 234 allows the user to return to the original interestarea of the multimedia asset collection 200 (FIG. 2) after exploring theside-track. In addition to the updated spiral formation of multimediaassets 232 a new selection of random multimedia assets 224 is alsopresented on display screen 220 according to an embodiment of thepresent invention.

Referring now to FIG. 6 a graphical representation is shown depicting analternate embodiment of the present invention where the rank-orderedmultimedia assets are arranged using a matrix arrangement of multimediaassets 236. The highest ranked multimedia asset 237 is positioned in theupper left corner of the matrix arrangement of multimedia assets 236.Lower-ranked multimedia assets are displayed proceeding fromleft-to-right across the top row of the matrix arrangement, and onsubsequent rows. A selection of random multimedia assets 238 aredispersed around the periphery of the rank-ordered assets on displayscreen 220 according to one embodiment of the present invention. In thisexample, the random multimedia assets 238 are presented slightly skewedto intuitively indicate that they are randomly selected assets. Anoptional scrolling feature (not shown) can also be used to scrollthrough the ranked multimedia asset collection 206 (FIG. 2), enablingthe user to view more images from large multimedia asset collections.

Turning now to FIG. 7, a graphical representation is shown depicting analternate embodiment of the present invention where the rank-orderedmultimedia assets 246 are displayed on display screen 220 using a “filmstrip” preview selector 240. The preview selector 240 includesdirectional user interface indicators 242 to scroll through themultimedia asset collection. A preview window icon 244 indicates thecurrently selected multimedia asset of interest. A previewed multimediaasset 248 is shown within the preview window icon 244 and is also shownin magnified form as magnified multimedia asset 250. In this example,the multimedia assets adjacent to the previewed multimedia asset 248 arealso shown in magnified form. As a user interacts with the directionaluser interface indicators 242, the “film strip” containing the previewedmultimedia asset 248 is scrolled through the ranked multimedia assetcollection 206 (FIG. 2) and the magnified multimedia asset 250 isupdated accordingly.

Optionally, random multimedia assets 252 can be presented adjacent tothe preview selector 240. In this example, the random multimedia assets252 are presented in a slightly skewed format to intuitively indicatethat they are random assets. The random multimedia assets 252 alsorespond to the directional user interface indicators 242 and scrollalong with the previewed multimedia asset 248 and the magnifiedmultimedia asset 250.

Referring now to FIG. 8, a graphical representation is shown depictingrank-ordered multimedia assets 256 in a rank-ordered list arrangement254 on display screen 220 according to an alternate embodiment of thepresent invention. In the rank-ordered list arrangement 254, associatedmetadata 258 is presented adjacent to the rank-ordered multimedia assets256 in order to provide additional information to the user regarding therank-ordered multimedia assets 256. The associated metadata 258 caninclude information such as the file name, capture time/date, capturelocation, file size and file location. It will be obvious to one skilledin the art that many other types of associated metadata could also bedisplayed in accordance with the present invention. Also, to indicatethat the rank-ordered multimedia assets 256 and associated metadata 258are part of the list arrangement 254, a rank-ordered asset indicator 260is included as a visible border surrounding the rank-ordered listarrangement 254. Optionally, random multimedia assets 262 can bepresented adjacent to rank-ordered list arrangement 254 together withassociated metadata 264. In addition, a random asset indicator 266 isshown as an alternative border style. In this example, the random assetindicator 266 surrounds only individual assets unlike rank-ordered assetindicator 260 which surrounds the entire visible portion of the list.Also included is a list scroll control user interface 268 which controlsboth the ranked ordered and random assets.

The spiral, matrix, film strip, and list presentation formats shown inFIGS. 3-8 represent examples of user selectable interaction options. Itwill be obvious to one skilled in the art that many other types of userinterfaces can also be used in accordance with the present invention.

The present invention can be implemented as a computer program productand can include one or more storage medium, for example; magneticstorage media such as magnetic disk (such as a floppy disk) or magnetictape; optical storage media such as optical disk, optical tape, ormachine readable bar code; solid-state electronic storage devices suchas random access memory (RAM), or read-only memory (ROM); or any otherphysical device or media employed to store a computer program havinginstructions for controlling one or more computers to practice themethod according to the present invention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   110 data processing system-   120 peripheral system-   130 user interface system-   140 data storage system-   200 multimedia asset collection-   202 interest criterion-   204 rank multimedia assets step-   206 ranked multimedia asset collection step-   208 display portion of multimedia collection step-   210 displayed multimedia assets step-   212 select multimedia asset step-   214 selected multimedia asset step-   216 update interest criterion step-   218 updated interest criterion step-   220 display screen-   222 initial spiral formation of multimedia assets-   223 highest-ranked multimedia asset-   224 random multimedia assets-   226 selected multimedia asset-   228 updated spiral formation of multimedia assets-   230 selected multimedia asset-   232 updated spiral formation of multimedia assets-   234 maintained multimedia assets-   236 matrix arrangement of multimedia assets-   237 highest-ranked multimedia asset-   238 random multimedia assets-   240 preview selector-   242 directional user interface indicator-   244 preview window icon-   246 rank-ordered multimedia assets-   248 previewed multimedia asset-   250 magnified multimedia asset-   252 random multimedia assets-   254 rank-ordered list arrangement-   256 rank-ordered multimedia assets-   258 associated metadata-   260 rank-ordered asset indicator-   262 random multimedia assets-   264 associated metadata-   266 random asset indicator-   268 list scroll control user interface

The invention claimed is:
 1. A non-transitory computer readable mediumencoded with a computer program, the program comprising instructionsthat when executed by a data processing device cause the data processingdevice to: identify a collection of multimedia assets, each multimediaasset having metadata descriptive of the multimedia asset's content;receive a user-defined interest criterion that corresponds to, at leastin part, one or more factors relating to the metadata; identify a firstportion of multimedia assets from the collection of multimedia assets,the first portion comprising multimedia assets, wherein inclusion withinthe first portion is based, at least in part, on the interest criterion;identify a second portion of multimedia assets from the collection ofmultimedia assets, the second portion comprising multimedia assets thatare not related to the interest criterion; rank, using a processor, themedia assets in the first portion of multimedia assets based at least inpart on the interest criterion; display, in ranking order, at least someof the first portion of multimedia assets in a first region of adisplay; display at least some of the second portion of multimediaassets in a second region of the display, wherein the first and secondregions are disjoint; select one of the displayed multimedia assets;determine an updated interest criterion based, at least in part, on theinterest criterion and metadata of the selected multimedia asset;determine a new ranking, using the data processing device, of themultimedia assets based at least in part on the updated interestcriterion; identify a new first portion of multimedia assets from thecollection of multimedia assets based, at least in part, on the updatedinterest criterion; identify a new second portion of multimedia assetsfrom the collection of multimedia assets that are not related to theinterest criterion; display, in new ranking order, at least some of thenew first portion of multimedia assets in a new first region of adisplay; and display at least some of the new second portion ofmultimedia assets in a new second region of the display, wherein the newfirst and new second regions are disjoint.
 2. The non-transitory storagedevice of claim 1, wherein the instructions to cause the data processingdevice to select one of the multimedia assets comprise instructions tocause the data processing device to select one of the multimedia assetsfrom the first portion.
 3. The non-transitory storage device of claim 1,wherein the instructions to cause the data processing device to selectone of the multimedia assets comprise instructions to cause the dataprocessing device to select one of the multimedia assets from the secondportion.
 4. The non-transitory storage device of claim 3, wherein theinterest criterion carries no weight in defining the updated interestcriterion.
 5. The non-transitory storage device of claim 1, wherein themetadata comprises semantic metadata determined by automaticallyanalyzing the multimedia assets using a semantic analysis algorithm. 6.The non-transitory storage device of claim 1, wherein the instructionsto cause the data processing device to display, in ranking order, atleast some of the first portion of multimedia assets in a first regionof a display comprise instructions to cause the data processing deviceto adjust the size or position of the displayed multimedia assets based,at least in part, on the ranking.
 7. The non-transitory storage deviceof claim 6, wherein the instructions to cause the data processingapparatus to adjust the size or position of the displayed multimediaassets based, at least in part, on the ranking comprise instructions tocause the data processing apparatus to position, in an ordered sequence,the displayed assets, wherein the highest ranked multimedia assets aredisplayed at the start of the ordered sequence.
 8. The non-transitorystorage device of claim 7, wherein the ordered sequence is a spiral ofoverlapping images.
 9. The non-transitory storage device of claim 7,wherein the ordered sequence comprises a matrix arrangement ofmultimedia assets having one or more rows.
 10. A method for browsing acollection of multimedia assets on a display comprising: identifying acollection of multimedia assets, each multimedia asset having metadatadescriptive of the multimedia asset's content; receiving an interestcriterion that corresponds to, at least in part, one or more factorsrelating to the metadata; identifying a first portion of multimediaassets from the collection of multimedia assets the first portioncomprising multimedia assets, wherein inclusion within the first portionis based, at least in part, on the interest criterion; identifying asecond portion of multimedia assets from the collection of multimediaassets that are not related to the interest criterion; ranking, using aprocessor, the media assets in the first portion of multimedia assetsbased at least in part on the interest criterion; displaying, in rankingorder, at least some of the first portion of multimedia assets in afirst region of a display; displaying at least some of the secondportion of multimedia assets in a second region of the display, whereinthe first and second regions are disjoint; selecting one of thedisplayed multimedia assets; determining an updated interest criterionbased, at least in part, on the interest criterion and metadata of theselected multimedia asset; determining a new ranking, using the dataprocessing device, of the multimedia assets based at least in part onthe updated interest criterion; identifying a new first portion ofmultimedia assets from the collection of multimedia assets based, atleast in part, on the updated interest criterion; identifying a newsecond portion of multimedia assets from the collection of multimediaassets that are not related to the interest criterion; displaying, innew ranking order, at least some of the new first portion of multimediaassets in a new first region of a display; and displaying at least someof the new second portion of multimedia assets in a new second region ofthe display, wherein the new first and new second regions are disjoint.11. The method of claim 10, wherein the selected multimedia asset is amultimedia asset from the first portion.
 12. The method of claim 10,wherein the selected multimedia asset is a multimedia asset from thesecond portion.
 13. The method of claim 12, wherein the interestcriterion carries no weight in defining the updated interest criterion.14. The method of claim 10, wherein the metadata comprises semanticmetadata determined by automatically analyzing the multimedia assetsusing a semantic analysis algorithm.
 15. The method of claim 10, furthercomprising adjusting the size or position of the displayed multimediaassets based, at least in part, on the ranking.
 16. The method of claim15, further comprising positioning, in an ordered sequence, thedisplayed assets, wherein the highest ranked multimedia assets aredisplayed at the start of the ordered sequence.
 17. The method of claim15, wherein the ordered sequence is a spiral of overlapping images. 18.A system comprising: a display; and one or more processors configured toperform operations comprising: identifying a collection of multimediaassets, each multimedia asset having metadata descriptive of themultimedia asset's content; receiving an interest criterion thatcorresponds to, at least in part, one or more factors relating to themetadata; identifying a first portion of multimedia assets from thecollection of multimedia assets, the first portion comprising multimediaassets, wherein inclusion within the first portion is based, at least inpart, on the interest criterion; identifying a second portion ofmultimedia assets from the collection of multimedia assets that are notrelated to the interest criterion; ranking, using a processor, the mediaassets in the first portion of multimedia assets based at least in parton the interest criterion; displaying, in ranking order, at least someof the first portion of multimedia assets in a first region of adisplay; displaying at least some of the second portion of multimediaassets in a second region of the display, wherein the first and secondregions are disjoint; selecting one of the displayed multimedia assets;determining an updated interest criterion based, at least in part, onthe interest criterion and metadata of the selected multimedia asset;determining a new ranking, using the data processing device, of themultimedia assets based at least in part on the updated interestcriterion; identifying a new first portion of multimedia assets from thecollection of multimedia assets based, at least in part, on the updatedinterest criterion; identifying a new second portion of multimediaassets from the collection of multimedia assets that are not related tothe interest criterion; displaying, in new ranking order, at least someof the new first portion of multimedia assets in a new first region of adisplay; and displaying at least some of the new second portion ofmultimedia assets in a new second region of the display, wherein the newfirst and new second regions are disjoint.